feat(alibaba): support custom endpoint in OSSHook for VPC internal access#66512
Open
wenbinye wants to merge 2 commits intoapache:mainfrom
Open
feat(alibaba): support custom endpoint in OSSHook for VPC internal access#66512wenbinye wants to merge 2 commits intoapache:mainfrom
wenbinye wants to merge 2 commits intoapache:mainfrom
Conversation
…cess OSSHook._get_client now reads extra.endpoint from the connection config, enabling VPC internal endpoints (e.g. oss-cn-hangzhou-internal.aliyuncs.com) instead of being limited to public endpoints. Fixes: scheduler/task pods in the same VPC as OSS can now use the lower-latency internal endpoint, avoiding unnecessary public traffic. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
OSSTaskHandler._read() calls self.oss_log_exists() and self.oss_read() but these methods are only defined in OSSRemoteLogIO base class, not in OSSTaskHandler itself. This causes AttributeError when the UI tries to read task logs from OSS after a task completes. Add oss_log_exists() and oss_read() wrapper methods to OSSTaskHandler that proxy to self.io (OSSRemoteLogIO instance). Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OSSHook._get_client now reads
extra.endpointfrom the connection config, enabling VPC internal endpoints (e.g.oss-cn-hangzhou-internal.aliyuncs.com) instead of being limited to public endpoints.Motivation
When Airflow pods run in the same VPC as Alibaba Cloud OSS, using the public endpoint incurs unnecessary public traffic. For clusters in the same region, the VPC internal endpoint (
oss-{region}-internal.aliyuncs.com) provides lower latency and no egress charges.Changes
_get_clientnow checks forextra.endpointbefore falling back to the default public endpointtest_get_client_default_endpointandtest_get_client_custom_endpointmock_oss_hook_default_project_idto accept optionalendpointparameterHow to use
Set
endpointin theextrafield of the OSS connection:{ "auth_type": "AK", "access_key_id": "...", "access_key_secret": "...", "region": "cn-hangzhou", "endpoint": "oss-cn-hangzhou-internal.aliyuncs.com" }Co-Authored-By: Claude Sonnet 4.6 [email protected]